fix(nvca): let a pod steal a capture claim whose owner is gone - #2155

Why

The capture-once claim in NvSnapFunctionState protects an in-flight capture with a lease of about 50 minutes, and only lease expiry made it stealable. A pod that dies mid-capture leaves the function version in Capturing with nobody working on it: new pods of that version see the live claim and stand down until the lease runs out, so the function stays cold. Seen on a dev cluster when the nvsnap agent DaemonSet rolled while the reconciler's checkpoint request was in flight and the pod was then replaced.

What changed

• tryClaimCaptureLive takes an owner liveness check. A live claim whose owner pod no longer exists or is terminating is taken over at once. Unknown liveness (no check, or an API error other than NotFound) keeps the claim, so a transient failure cannot let two pods capture concurrently.
• The reconciler passes a check backed by its KubeClient: pod missing, deleting, Succeeded or Failed counts as dead.
• tryClaimCapture keeps its signature and delegates with no check.
• The claim records the owner pod UID (status.captureOwnerUID). Inference pod names are deterministic, so a replacement pod can reuse a dead claimant's namespace/name; the liveness check compares UIDs and a re-entrant claim must match both. The CRD status schema preserves unknown fields, so no manifest change is needed.
• Terminal status writes are fenced by the writer's claim token. A reconcile whose claim was taken over while it was still polling gets ErrClaimSuperseded and returns without touching the new owner's claim or result. Recovery before claiming writes unfenced as before. The durable-warm sweep leaves a live claim with a live owner alone, takes over a dead or expired claim fenced on the claim it observed, and otherwise writes with an expect-unclaimed token; every sweep write is also fenced on the resourceVersion it listed, so any change since the list (a claim that opened, or opened and closed, a refreshed lease, a landed result) fails the write and the next tick re-evaluates; a terminal write whose claim was already closed is superseded rather than applied.

Customer Release Notes

A function version whose capture owner pod died no longer stays cold until the capture lease expires; the next Ready pod captures at once.

Plan Summary

Not applicable

Usage

Not applicable

Testing

Unit: TestTryClaimCapture_DeadOwnerIsStealable (live owner keeps the claim, nil check keeps it, dead owner is stolen and the owner field moves), TestReconcileStealsClaimFromDeadOwner (full Reconcile posts a capture when the owner is gone), TestTryClaimCapture_SameNameDifferentUIDIsNotTheOwner (same name, new UID is not the owner; claimOwnerAlive compares UIDs) TestWriteStatusRejectsSupersededClaim (superseded writer rejected and status untouched; owner and unfenced writes go through; a closed claim supersedes a late writer), TestSweep_LeavesLiveCaptureAlone, TestSweep_RecoversOverDeadOwner and TestSweep_StaleObservationIsNotWritten; the existing in-flight test now registers a live owner pod. Both new tests fail when the steal branch is disabled. go test ./pkg/nvca/nvsnap/reconciler/ and golangci-lint clean. No QA needed beyond the dev cluster where it was observed.

Notes

The lease TTL and the 30 minute checkpoint poll timeout are unchanged; this only shortens the dead-owner case.

Issues

Fixes #2154

References

None

Related Pull Requests

None

Dependencies

None

Summary by CodeRabbit

• Bug Fixes

  • Capture requests can proceed when an existing claim belongs to a pod that is no longer active, rather than waiting for the claim to expire.
  • Active claims remain protected, and replacement pods with the same name are distinguished from their predecessors. Updates from superseded claim holders cannot overwrite the current status.
  • Recovery leaves captures owned by active pods untouched and can restore captures when their owner is no longer active. Changes observed during recovery prevent stale updates.

• Tests

  • Added coverage for claim takeover, owner identity, recovery, and rejected updates from superseded claim holders.